Image data encoding and decoding

ABSTRACT

An image decoding apparatus comprises a selector configured to select, from a set of candidate prediction operations each defining at least a prediction direction, a prediction operation for prediction of samples of a current region of a current image, the current region comprising an array of two or more rows and two or more columns of samples; and an intra-image predictor configured to derive predicted samples of the current region with respect to one or more of a group of reference samples of the same image in dependence upon a prediction direction, defined by the selected prediction operation, between a current sample to be predicted and a reference position amongst the reference samples; in which, for at least some of the candidate prediction operations, the group of reference samples comprises two or more parallel linear arrays of reference samples disposed at different respective separations from the current region.

BACKGROUND Reference to Earlier Applications

This application claims priority from GB1717684.3 filed on 27 Oct. 2017and GB1812556.7 filed on 1 Aug. 2018. The contents of both of thesepriority applications are hereby incorporated by reference.

Field

This disclosure relates to image data encoding and decoding.

Description of Related Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, is neitherexpressly or impliedly admitted as prior art against the presentdisclosure.

There are several video data encoding and decoding systems which involvetransforming video data into a frequency domain representation,quantising the frequency domain coefficients and then applying some formof entropy encoding to the quantised coefficients. This can achievecompression of the video data. A corresponding decoding or decompressiontechnique is applied to recover a reconstructed version of the originalvideo data.

Current video codecs (coder-decoders) such as those used in H.264/MPEG-4Advanced Video Coding (AVC) achieve data compression primarily by onlyencoding the differences between successive video frames. These codecsuse a regular array of so-called macroblocks, each of which is used as aregion of comparison with a corresponding macroblock in a previous videoframe, and the image region within the macroblock is then encodedaccording to the degree of motion found between the correspondingcurrent and previous macroblocks in the video sequence, or betweenneighbouring macroblocks within a single frame of the video sequence.

High Efficiency Video Coding (HEVC), also known as H.265 or MPEG-H Part2, is a proposed successor to H.264/MPEG-4 AVC. It is intended for HEVCto improve video quality and double the data compression ratio comparedto H.264, and for it to be scalable from 128×96 to 7680×4320 pixelsresolution, roughly equivalent to bit rates ranging from 128 kbit/s to800 Mbit/s.

SUMMARY

The present disclosure addresses or mitigates problems arising from thisprocessing.

Respective aspects and features of the present disclosure are defined inthe appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary, but are notrestrictive, of the present technology.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 schematically illustrates an audio/video (AN) data transmissionand reception system using video data compression and decompression;

FIG. 2 schematically illustrates a video display system using video datadecompression;

FIG. 3 schematically illustrates an audio/video storage system usingvideo data compression and decompression;

FIG. 4 schematically illustrates a video camera using video datacompression;

FIGS. 5 and 6 schematically illustrate storage media;

FIG. 7 provides a schematic overview of a video data compression anddecompression apparatus;

FIG. 8 schematically illustrates a predictor;

FIG. 9 schematically illustrates a partially-encoded image;

FIG. 10 schematically illustrates a set of possible intra-predictiondirections;

FIG. 11 schematically illustrates a set of prediction modes;

FIG. 12 schematically illustrates another set of prediction modes;

FIG. 13 schematically illustrates an intra-prediction process;

FIGS. 14 and 15 schematically illustrate a reference sample projectionprocess;

FIG. 16 schematically illustrates a predictor;

FIGS. 17 and 18 schematically illustrate the use of projected referencesamples;

FIG. 19 schematically illustrates a prediction process;

FIGS. 20 to 22 schematically illustrate example interpolationtechniques;

FIGS. 23 to 26 schematically illustrate respective groups of rows andcolumns of reference samples;

FIGS. 27 to 30 schematically represent respective projected versions ofFIGS. 23 to 26;

FIG. 31 schematically represents an intra mode selector;

FIGS. 32 and 33 are respective schematic flowcharts representing methodsof operation of the intra mode selector of FIG. 31;

FIG. 34 schematically represents an intra mode selector;

FIGS. 35 and 36 are respective schematic flowcharts representing methodsof operation of an intra predictor;

FIG. 37 schematically illustrate a part of the functionality of an intrapredictor;

FIG. 38 is a schematic flowchart representing operations of thearrangement of FIG. 37;

FIGS. 39 and 40 are schematic flowcharts illustrating respectivemethods;

FIGS. 41 to 44 schematically illustrate sets of linear arrays;

FIG. 45 schematically represents an intra mode selector;

FIG. 46 is a schematic flowchart illustrating method steps; and

FIGS. 47 and 48 schematically illustrate sets of linear arrays.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, FIGS. 1-4 are provided to give schematicillustrations of apparatus or systems making use of the compressionand/or decompression apparatus to be described below in connection withembodiments of the present technology.

All of the data compression and/or decompression apparatus to bedescribed below may be implemented in hardware, in software running on ageneral-purpose data processing apparatus such as a general-purposecomputer, as programmable hardware such as an application specificintegrated circuit (ASIC) or field programmable gate array (FPGA) or ascombinations of these. In cases where the embodiments are implemented bysoftware and/or firmware, it will be appreciated that such softwareand/or firmware, and non-transitory data storage media by which suchsoftware and/or firmware are stored or otherwise provided, areconsidered as embodiments of the present technology.

FIG. 1 schematically illustrates an audio/video data transmission andreception system using video data compression and decompression.

An input audio/video signal 10 is supplied to a video data compressionapparatus 20 which compresses at least the video component of theaudio/video signal 10 for transmission along a transmission route 30such as a cable, an optical fibre, a wireless link or the like. Thecompressed signal is processed by a decompression apparatus 40 toprovide an output audio/video signal 50. For the return path, acompression apparatus 60 compresses an audio/video signal fortransmission along the transmission route 30 to a decompressionapparatus 70.

The compression apparatus 20 and decompression apparatus 70 cantherefore form one node of a transmission link. The decompressionapparatus 40 and decompression apparatus 60 can form another node of thetransmission link. Of course, in instances where the transmission linkis uni-directional, only one of the nodes would require a compressionapparatus and the other node would only require a decompressionapparatus.

FIG. 2 schematically illustrates a video display system using video datadecompression. In particular, a compressed audio/video signal 100 isprocessed by a decompression apparatus 110 to provide a decompressedsignal which can be displayed on a display 120. The decompressionapparatus 110 could be implemented as an integral part of the display120, for example being provided within the same casing as the displaydevice. Alternatively, the decompression apparatus 110 may be providedas (for example) a so-called set top box (STB), noting that theexpression “set-top” does not imply a requirement for the box to besited in any particular orientation or position with respect to thedisplay 120; it is simply a term used in the art to indicate a devicewhich is connectable to a display as a peripheral device.

FIG. 3 schematically illustrates an audio/video storage system usingvideo data compression and decompression. An input audio/video signal130 is supplied to a compression apparatus 140 which generates acompressed signal for storing by a store device 150 such as a magneticdisk device, an optical disk device, a magnetic tape device, a solidstate storage device such as a semiconductor memory or other storagedevice. For replay, compressed data is read from the storage device 150and passed to a decompression apparatus 160 for decompression to providean output audio/video signal 170.

It will be appreciated that the compressed or encoded signal, and astorage medium such as a machine-readable non-transitory storage medium,storing that signal, are considered as embodiments of the presenttechnology.

FIG. 4 schematically illustrates a video camera using video datacompression. In FIG. 4, an image capture device 180, such as a chargecoupled device (CCD) image sensor and associated control and read-outelectronics, generates a video signal which is passed to a compressionapparatus 190. A microphone (or plural microphones) 200 generates anaudio signal to be passed to the compression apparatus 190. Thecompression apparatus 190 generates a compressed audio/video signal 210to be stored and/or transmitted (shown generically as a schematic stage220).

The techniques to be described below relate primarily to video datacompression and decompression. It will be appreciated that many existingtechniques may be used for audio data compression in conjunction withthe video data compression techniques which will be described, togenerate a compressed audio/video signal. Accordingly, a separatediscussion of audio data compression will not be provided. It will alsobe appreciated that the data rate associated with video data, inparticular broadcast quality video data, is generally very much higherthan the data rate associated with audio data (whether compressed oruncompressed). It will therefore be appreciated that uncompressed audiodata could accompany compressed video data to form a compressedaudio/video signal. It will further be appreciated that although thepresent examples (shown in FIGS. 1-4) relate to audio/video data, thetechniques to be described below can find use in a system which simplydeals with (that is to say, compresses, decompresses, stores, displaysand/or transmits) video data. That is to say, the embodiments can applyto video data compression without necessarily having any associatedaudio data handling at all.

FIG. 4 therefore provides an example of a video capture apparatuscomprising an image sensor and an encoding apparatus of the type to bediscussed below. FIG. 2 therefore provides an example of a decodingapparatus of the type to be discussed below and a display to which thedecoded images are output.

A combination of FIGS. 2 and 4 may provide a video capture apparatuscomprising an image sensor 180 and encoding apparatus 190, decodingapparatus 110 and a display 120 to which the decoded images are output.

FIGS. 5 and 6 schematically illustrate storage media, which store (forexample) the compressed data generated by the apparatus 20, 60, thecompressed data input to the apparatus 110 or the storage media orstages 150, 220. FIG. 5 schematically illustrates a disc storage mediumsuch as a magnetic or optical disc, and FIG. 6 schematically illustratesa solid state storage medium such as a flash memory. Note that FIGS. 5and 6 can also provide examples of non-transitory machine-readablestorage media which store computer software which, when executed by acomputer, causes the computer to carry out one or more of the methods tobe discussed below.

Therefore, the above arrangements provide examples of video storage,capture, transmission or reception apparatuses embodying any of thepresent techniques.

FIG. 7 provides a schematic overview of a video data compression anddecompression apparatus.

A controller 343 controls the overall operation of the apparatus and, inparticular when referring to a compression mode, controls a trialencoding processes by acting as a selector to select various modes ofoperation such as block sizes and shapes, and whether the video data isto be encoded losslessly or otherwise. The controller is considered topart of the image encoder or image decoder (as the case may be).Successive images of an input video signal 300 are supplied to an adder310 and to an image predictor 320. The image predictor 320 will bedescribed below in more detail with reference to FIG. 8. The imageencoder or decoder (as the case may be) plus the intra-image predictorof FIG. 8 may use features from the apparatus of FIG. 7. This does notmean that the image encoder or decoder necessarily requires everyfeature of FIG. 7 however.

The adder 310 in fact performs a subtraction (negative addition)operation, in that it receives the input video signal 300 on a “+” inputand the output of the image predictor 320 on a “−” input, so that thepredicted image is subtracted from the input image. The result is togenerate a so-called residual image signal 330 representing thedifference between the actual and projected images.

One reason why a residual image signal is generated is as follows. Thedata coding techniques to be described, that is to say the techniqueswhich will be applied to the residual image signal, tend to work moreefficiently when there is less “energy” in the image to be encoded.Here, the term “efficiently” refers to the generation of a small amountof encoded data; for a particular image quality level, it is desirable(and considered “efficient”) to generate as little data as ispracticably possible. The reference to “energy” in the residual imagerelates to the amount of information contained in the residual image. Ifthe predicted image were to be identical to the real image, thedifference between the two (that is to say, the residual image) wouldcontain zero information (zero energy) and would be very easy to encodeinto a small amount of encoded data. In general, if the predictionprocess can be made to work reasonably well such that the predictedimage content is similar to the image content to be encoded, theexpectation is that the residual image data will contain lessinformation (less energy) than the input image and so will be easier toencode into a small amount of encoded data.

The remainder of the apparatus acting as an encoder (to encode theresidual or difference image) will now be described. The residual imagedata 330 is supplied to a transform unit or circuitry 340 whichgenerates a discrete cosine transform (DCT) representation of blocks orregions of the residual image data. The DCT technique itself is wellknown and will not be described in detail here. Note also that the useof DCT is only illustrative of one example arrangement. Other transformswhich might be used include, for example, the discrete sine transform(DST). A transform could also comprise a sequence or cascade ofindividual transforms, such as an arrangement in which one transform isfollowed (whether directly or not) by another transform. The choice oftransform may be determined explicitly and/or be dependent upon sideinformation used to configure the encoder and decoder.

The output of the transform unit 340, which is to say, a set of DCTcoefficients for each transformed block of image data, is supplied to aquantiser 350. Various quantisation techniques are known in the field ofvideo data compression, ranging from a simple multiplication by aquantisation scaling factor through to the application of complicatedlook-up tables under the control of a quantisation parameter. Thegeneral aim is twofold. Firstly, the quantisation process reduces thenumber of possible values of the transformed data. Secondly, thequantisation process can increase the likelihood that values of thetransformed data are zero. Both of these can make the entropy encodingprocess, to be described below, work more efficiently in generatingsmall amounts of compressed video data.

A data scanning process is applied by a scan unit 360. The purpose ofthe scanning process is to reorder the quantised transformed data so asto gather as many as possible of the non-zero quantised transformedcoefficients together, and of course therefore to gather as many aspossible of the zero-valued coefficients together. These features canallow so-called run-length coding or similar techniques to be appliedefficiently. So, the scanning process involves selecting coefficientsfrom the quantised transformed data, and in particular from a block ofcoefficients corresponding to a block of image data which has beentransformed and quantised, according to a “scanning order” so that (a)all of the coefficients are selected once as part of the scan, and (b)the scan tends to provide the desired reordering. One example scanningorder which can tend to give useful results is a so-called up-rightdiagonal scanning order.

The scanned coefficients are then passed to an entropy encoder (EE) 370.Again, various types of entropy encoding may be used. Two examples arevariants of the so-called CABAC (Context Adaptive Binary ArithmeticCoding) system and variants of the so-called CAVLC (Context AdaptiveVariable-Length Coding) system. In general terms, CABAC is considered toprovide a better efficiency, and in some studies has been shown toprovide a 10-20% reduction in the quantity of encoded output data for acomparable image quality compared to CAVLC. However, CAVLC is consideredto represent a much lower level of complexity (in terms of itsimplementation) than CABAC. Note that the scanning process and theentropy encoding process are shown as separate processes, but in factcan be combined or treated together. That is to say, the reading of datainto the entropy encoder can take place in the scan order. Correspondingconsiderations apply to the respective inverse processes to be describedbelow.

The output of the entropy encoder 370, along with additional data(mentioned above and/or discussed below), for example defining themanner in which the predictor 320 generated the predicted image,provides a compressed output video signal 380.

However, a return path is also provided because the operation of thepredictor 320 itself depends upon a decompressed version of thecompressed output data.

The reason for this feature is as follows. At the appropriate stage inthe decompression process (to be described below) a decompressed versionof the residual data is generated. This decompressed residual data hasto be added to a predicted image to generate an output image (becausethe original residual data was the difference between the input imageand a predicted image). In order that this process is comparable, asbetween the compression side and the decompression side, the predictedimages generated by the predictor 320 should be the same during thecompression process and during the decompression process. Of course, atdecompression, the apparatus does not have access to the original inputimages, but only to the decompressed images. Therefore, at compression,the predictor 320 bases its prediction (at least, for inter-imageencoding) on decompressed versions of the compressed images.

The entropy encoding process carried out by the entropy encoder 370 isconsidered (in at least some examples) to be “lossless”, which is to saythat it can be reversed to arrive at exactly the same data which wasfirst supplied to the entropy encoder 370. So, in such examples thereturn path can be implemented before the entropy encoding stage.Indeed, the scanning process carried out by the scan unit 360 is alsoconsidered lossless, but in the present embodiment the return path 390is from the output of the quantiser 350 to the input of a complimentaryinverse quantiser 420. In instances where loss or potential loss isintroduced by a stage, that stage may be included in the feedback loopformed by the return path. For example, the entropy encoding stage canat least in principle be made lossy, for example by techniques in whichbits are encoded within parity information. In such an instance, theentropy encoding and decoding should form part of the feedback loop.

In general terms, an entropy decoder 410, the reverse scan unit 400, aninverse quantiser 420 and an inverse transform unit or circuitry 430provide the respective inverse functions of the entropy encoder 370, thescan unit 360, the quantiser 350 and the transform unit 340. For now,the discussion will continue through the compression process; theprocess to decompress an input compressed video signal will be discussedseparately below.

In the compression process, the scanned coefficients are passed by thereturn path 390 from the quantiser 350 to the inverse quantiser 420which carries out the inverse operation of the scan unit 360. An inversequantisation and inverse transformation process are carried out by theunits 420, 430 to generate a compressed-decompressed residual imagesignal 440.

The image signal 440 is added, at an adder 450, to the output of thepredictor 320 to generate a reconstructed output image 460. This formsone input to the image predictor 320, as will be described below.

Turning now to the process applied to decompress a received compressedvideo signal 470, the signal is supplied to the entropy decoder 410 andfrom there to the chain of the reverse scan unit 400, the inversequantiser 420 and the inverse transform unit 430 before being added tothe output of the image predictor 320 by the adder 450. So, at thedecoder side, the decoder reconstructs a version of the residual imageand then applies this (by the adder 450) to the predicted version of theimage (on a block by block basis) so as to decode each block. Instraightforward terms, the output 460 of the adder 450 forms the outputdecompressed video signal 480. In practice, further filtering mayoptionally be applied (for example, by a filter 560 shown in FIG. 8 butomitted from FIG. 7 for clarity of the higher level diagram of FIG. 7)before the signal is output.

The apparatus of FIGS. 7 and 8 can act as a compression (encoding)apparatus or a decompression (decoding) apparatus. The functions of thetwo types of apparatus substantially overlap. The scan unit 360 andentropy encoder 370 are not used in a decompression mode, and theoperation of the predictor 320 (which will be described in detail below)and other units follow mode and parameter information contained in thereceived compressed bit-stream rather than generating such informationthemselves.

FIG. 8 schematically illustrates the generation of predicted images, andin particular the operation of the image predictor 320.

There are two basic modes of prediction carried out by the imagepredictor 320: so-called intra-image prediction and so-calledinter-image, or motion-compensated (MC), prediction. At the encoderside, each involves detecting a prediction direction in respect of acurrent block to be predicted, and generating a predicted block ofsamples according to other samples (in the same (intra) or another(inter) image). By virtue of the units 310 or 450, the differencebetween the predicted block and the actual block is encoded or appliedso as to encode or decode the block respectively.

(At the decoder, or at the reverse decoding side of the encoder, thedetection of a prediction direction may be in response to dataassociated with the encoded data by the encoder, indicating whichdirection was used at the encoder. Or the detection may be in responseto the same factors as those on which the decision was made at theencoder).

Intra-image prediction bases a prediction of the content of a block orregion of the image on data from within the same image. This correspondsto so-called I-frame encoding in other video compression techniques. Incontrast to I-frame encoding, however, which involves encoding the wholeimage by intra-encoding, in the present embodiments the choice betweenintra- and inter-encoding can be made on a block-by-block basis, thoughin other embodiments the choice is still made on an image-by-imagebasis.

Motion-compensated prediction is an example of inter-image predictionand makes use of motion information which attempts to define the source,in another adjacent or nearby image, of image detail to be encoded inthe current image. Accordingly, in an ideal example, the contents of ablock of image data in the predicted image can be encoded very simply asa reference (a motion vector) pointing to a corresponding block at thesame or a slightly different position in an adjacent image.

A technique known as “block copy” prediction is in some respects ahybrid of the two, as it uses a vector to indicate a block of samples ata position displaced from the currently predicted block within the sameimage, which should be copied to form the currently predicted block.

Returning to FIG. 8, two image prediction arrangements (corresponding tointra- and inter-image prediction) are shown, the results of which areselected by a multiplexer 500 under the control of a mode signal 510(for example, from the controller 343) so as to provide blocks of thepredicted image for supply to the adders 310 and 450. The choice is madein dependence upon which selection gives the lowest “energy” (which, asdiscussed above, may be considered as information content requiringencoding), and the choice is signalled to the decoder within the encodedoutput data-stream. Image energy, in this context, can be detected, forexample, by carrying out a trial subtraction of an area of the twoversions of the predicted image from the input image, squaring eachpixel value of the difference image, summing the squared values, andidentifying which of the two versions gives rise to the lower meansquared value of the difference image relating to that image area. Inother examples, a trial encoding can be carried out for each selectionor potential selection, with a choice then being made according to thecost of each potential selection in terms of one or both of the numberof bits required for encoding and distortion to the picture.

The actual prediction, in the intra-encoding system, is made on thebasis of image blocks received as part of the signal 460, which is tosay, the prediction is based upon encoded-decoded image blocks in orderthat exactly the same prediction can be made at a decompressionapparatus. However, data can be derived from the input video signal 300by an intra-mode selector 520 to control the operation of theintra-image predictor 530.

For inter-image prediction, a motion compensated (MC) predictor 540 usesmotion information such as motion vectors derived by a motion estimator550 from the input video signal 300. Those motion vectors are applied toa processed version of the reconstructed image 460 by the motioncompensated predictor 540 to generate blocks of the inter-imageprediction.

Accordingly, the units 530 and 540 (operating with the estimator 550)each act as detectors to detect a prediction direction in respect of acurrent block to be predicted, and as a generator to generate apredicted block of samples (forming part of the prediction passed to theunits 310 and 450) according to other samples defined by the predictiondirection.

The processing applied to the signal 460 will now be described. Firstly,the signal is optionally filtered by a filter unit 560, which will bedescribed in greater detail below. This involves applying a “deblocking”filter to remove or at least tend to reduce the effects of theblock-based processing carried out by the transform unit 340 andsubsequent operations. A sample adaptive offsetting (SAO) filter mayalso be used. Also, an adaptive loop filter is optionally applied usingcoefficients derived by processing the reconstructed signal 460 and theinput video signal 300. The adaptive loop filter is a type of filterwhich, using known techniques, applies adaptive filter coefficients tothe data to be filtered. That is to say, the filter coefficients canvary in dependence upon various factors. Data defining which filtercoefficients to use is included as part of the encoded outputdata-stream.

The filtered output from the filter unit 560 in fact forms the outputvideo signal 480 when the apparatus is operating as a decompressionapparatus. It is also buffered in one or more image or frame stores 570;the storage of successive images is a requirement of motion compensatedprediction processing, and in particular the generation of motionvectors. To save on storage requirements, the stored images in the imagestores 570 may be held in a compressed form and then decompressed foruse in generating motion vectors. For this particular purpose, any knowncompression/decompression system may be used. The stored images arepassed to an interpolation filter 580 which generates a higherresolution version of the stored images; in this example, intermediatesamples (sub-samples) are generated such that the resolution of theinterpolated image is output by the interpolation filter 580 is 4 times(in each dimension) that of the images stored in the image stores 570for the luminance channel of 4:2:0 and 8 times (in each dimension) thatof the images stored in the image stores 570 for the chrominancechannels of 4:2:0. The interpolated images are passed as an input to themotion estimator 550 and also to the motion compensated predictor 540.

The way in which an image is partitioned for compression processing willnow be described. At a basic level, an image to be compressed isconsidered as an array of blocks or regions of samples. The splitting ofan image into such blocks or regions can be carried out by a decisiontree, such as that described in Bross et al: “High Efficiency VideoCoding (HEVC) text specification draft 6”, JCTVC-H1003_d0 (November2011), the contents of which are incorporated herein by reference. Insome examples, the resulting blocks or regions have sizes and, in somecases, shapes which, by virtue of the decision tree, can generallyfollow the disposition of image features within the image. This initself can allow for an improved encoding efficiency because samplesrepresenting or following similar image features would tend to begrouped together by such an arrangement. In some examples, square blocksor regions of different sizes (such as 4×4 samples up to, say, 64×64 orlarger blocks) are available for selection. In other examplearrangements, blocks or regions of different shapes such as rectangularblocks (for example, vertically or horizontally oriented) can be used.Other non-square and non-rectangular blocks are envisaged. The result ofthe division of the image into such blocks or regions is (in at leastthe present examples) that each sample of an image is allocated to one,and only one, such block or region.

The intra-prediction process will now be discussed. In general terms,intra-prediction involves generating a prediction of a current block ofsamples from previously-encoded and decoded samples in the same image.

FIG. 9 schematically illustrates a partially encoded image 800. Here,the image is being encoded from top-left to bottom-right on a block byblock basis. An example block encoded partway through the handling ofthe whole image is shown as a block 810. A shaded region 820 above andto the left of the block 810 has already been encoded. The intra-imageprediction of the contents of the block 810 can make use of any of theshaded area 820 but cannot make use of the unshaded area below that.

In some examples, the image is encoded on a block by block basis suchthat larger blocks (referred to as coding units or CUs) are encoded inan order such as the order discussed with reference to FIG. 9. Withineach CU, there is the potential (depending on the block splittingprocess that has taken place) for the CU to be handled as a set of twoor more smaller blocks or transform units (TUs). This can give ahierarchical order of encoding so that the image is encoded on a CU byCU basis, and each CU is potentially encoded on a TU by TU basis. Notehowever that for an individual TU within the current coding tree unit(the largest node in the tree structure of block division), thehierarchical order of encoding (CU by CU then TU by TU) discussed abovemeans that there may be previously encoded samples in the current CU andavailable to the coding of that TU which are, for example, above-rightor below-left of that TU.

The block 810 represents a CU; as discussed above, for the purposes ofintra-image prediction processing, this may be subdivided into a set ofsmaller units. An example of a current TU 830 is shown within the CU810. More generally, the picture is split into regions or groups ofsamples to allow efficient coding of signalling information andtransformed data. The signalling of the information may require adifferent tree structure of sub-divisions to that of the transform, andindeed that of the prediction information or the prediction itself. Forthis reason, the coding units may have a different tree structure tothat of the transform blocks or regions, the prediction blocks orregions and the prediction information. In some examples such as HEVCthe structure can be a so-called quad tree of coding units, whose leafnodes contain one or more prediction units and one or more transformunits; the transform units can contain multiple transform blockscorresponding to luma and chroma representations of the picture, andprediction could be considered to be applicable at the transform blocklevel. In examples, the parameters applied to a particular group ofsamples can be considered to be predominantly defined at a block level,which is potentially not of the same granularity as the transformstructure.

The intra-image prediction takes into account samples coded prior to thecurrent TU being considered, such as those above and/or to the left ofthe current TU. Source samples, from which the required samples arepredicted, may be located at different positions or directions relativeto the current TU. To decide which direction is appropriate for acurrent prediction unit, the mode selector 520 of an example encoder maytest all combinations of available TU structures for each candidatedirection and select the prediction direction and TU structure with thebest compression efficiency.

The picture may also be encoded on a “slice” basis. In one example, aslice is a horizontally adjacent group of CUs. But in more generalterms, the entire residual image could form a slice, or a slice could bea single CU, or a slice could be a row of CUs, and so on. Slices cangive some resilience to errors as they are encoded as independent units.The encoder and decoder states are completely reset at a slice boundary.For example, intra-prediction is not carried out across sliceboundaries; slice boundaries are treated as image boundaries for thispurpose.

FIG. 10 schematically illustrates a set of possible (candidate)prediction directions. The full set of candidate directions is availableto a prediction unit. The directions are determined by horizontal andvertical displacement relative to a current block position, but areencoded as prediction “modes”, a set of which is shown in FIG. 11. Notethat the so-called DC mode represents a simple arithmetic mean of thesurrounding upper and left-hand samples. Note also that the set ofdirections shown in FIG. 10 is just one example; in other examples, aset of (for example) 65 angular modes plus DC and planar (a full set of67 modes) as shown schematically in FIG. 12 makes up the full set. Othernumbers of modes could be used.

In general terms, after detecting a prediction direction, the systemsare operable to generate a predicted block of samples according to othersamples defined by the prediction direction. In examples, the imageencoder is configured to encode data identifying the predictiondirection selected for each sample or region of the image (and the imagedecoder is configured to detect such data).

FIG. 13 schematically illustrates an intra-prediction process in which asample 900 of a block or region 910 of samples is derived from otherreference samples 920 of the same image according to a direction 930defined by the intra-prediction mode associated with that sample. Thereference samples 920 in this example come from blocks above and to theleft of the block 910 in question and the predicted value of the sample900 is obtained by tracking along the direction 930 to the referencesamples 920. The direction 930 might point to a single individualreference sample but in a more general case an interpolated valuebetween surrounding reference samples is used as the prediction value.Note that the block 910 could be square as shown in FIG. 13 or could beanother shape such as rectangular.

FIGS. 14 and 15 schematically illustrate a previously proposed referencesample projection process.

In FIGS. 14 and 15, a block or region 1400 of samples to be predicted issurrounded by linear arrays of reference samples from which the intraprediction of the predicted samples takes place. The reference samples1410 are shown as shaded blocks in FIGS. 14 and 15, and the samples tobe predicted are shown as unshaded blocks. Note that an 8×8 block orregion of samples to be predicted is used in this example, but thetechniques are applicable to variable block sizes and indeed blockshapes.

As mentioned, the reference samples comprise at least two linear arraysin respective orientations with respect to the current image region ofsamples to be predicted. For example, the linear arrays may be an arrayor row 1420 of samples above the block of samples to be predicted and anarray or column 1430 of samples to the left of the block of samples tobe predicted.

As discussed above with reference to FIG. 13, the reference samplearrays can extend beyond the extent of the block to be predicted, inorder to provide for prediction modes or directions within the rangeindicated in FIGS. 10-12. Where necessary, if previously decoded samplesare not available for use as reference samples at particular referencesample positions, other reference samples can be re-used at thosemissing positions. Reference sample filtering processes can be used onthe reference samples.

A sample projection process is used to project at least some of thereference samples to different respective positions with respect to thecurrent image region, in the manner shown in FIGS. 14 and 15. In otherwords, in embodiments, the projection process and circuitry operates torepresent at least some of the reference samples at different spatialpositions relative to the current image region, for example as shown inFIGS. 14 and 15. Thus at least some reference samples may be moved (forthe purposes at least of defining an array of reference samples fromwhich samples are predicted) with respect to their relative positions tothe current image region. In particular, FIG. 14 relates to a projectionprocess performed for modes which are generally to the left of thediagonal mode (18 in FIG. 11) mainly modes 2 . . . 17, and FIG. 15schematically illustrates a reference sample projection carried formodes 19 . . . 34, namely those generally above the block to bepredicted (to the right of the diagonal mode 18). The diagonal mode 18can be assigned to either of these two groups as an arbitrary selection,such as to the group of modes to the right of the diagonal. So, in FIG.14, when the current prediction mode is between modes 2 and 17 (or theirequivalent in a system such as that of FIG. 12 having a different numberof possible prediction modes), the sample array 1420 above the currentblock is projected to form additional reference samples 1440 in the lefthand column. Prediction then takes place with respect to the linearprojected array 1450 formed of the original left hand column 1430 andthe projected samples 1440. In FIG. 15, for modes between 18 and 34 ofFIG. 11 (or their equivalent in other sets of prediction modes such asthose shown in FIG. 12), the reference samples 1500 in the left handcolumn are projected so as to extend to the left of the referencesamples 1510 above the current block. This forms a projected array 1520.

One reason why projection of this nature is carried out is to reduce thecomplexity of the intra prediction process, in that all of the samplesto be predicted are then referencing a single linear array of referencesamples, rather than referencing two orthogonal linear arrays.

FIG. 16 schematically illustrates a previously proposed predictioncircuitry 600 arranged to carry out the projection process of FIGS. 14and 15, specifically by providing projection circuitry 1610 configuredto carry out a projection process on the reference samples currentlyselected for a block of region to be predicted. The projected referencesamples are stored in a buffer 1620 to be accessed by an intra predictor1630 to generate predicted samples from the projected reference samples.The projection process is carried out according to the prediction modeassociated with the current block to be predicted, using the techniquesdiscussed in connection with FIGS. 14 and 15.

In embodiments, the same projection process is carried out in thedecoder and in the encoder, so that the predicted samples are the samein each instance.

Possible variations in operation between the use of prediction modeswhich will be referred to as “straight modes” and prediction modes whichwill be referred to as “curved modes” will now be discussed.

As further background, FIGS. 17 and 18 schematically illustrate anexample technique by which samples 1900 of a current region 1910 orblock to be predicted, are predicted from reference samples 1920. Inthis example, the reference samples have been projected into a lineararray using the techniques described with reference to FIGS. 14-16above.

A system of (x, y) coordinates is used for convenience, to allowindividual reference or predicted sample positions to be identified. Inthe example of FIG. 17, x coordinates are shown by a row 1930 ofnumbers, and y coordinates are shown by a column 1940 of numbers. So,each reference or predicted sample position has an associated (x, y)designation using the coordinate system.

In the example of FIG. 17, a generally vertical mode (for example, amode which is more vertical than horizontal) 1950, such as mode 23 inthe designation of FIG. 11, noting that a different mode number could beused if the set of modes shown in FIG. 12 were employed, has beenselected for prediction of samples 1900 of the block or region 1910. Asdiscussed above with reference to FIGS. 14-16, such a generally verticalprediction mode is handled by the circuitry of FIG. 16 by projecting theleft column of reference samples into an extension 1960 of the referencesamples above the block or region 1910.

Each of the samples to be predicted 1900 is predicted as follows. Foreach sample to be predicted, there is an associated (x, y) location suchas a location (0, 5) for a sample 1970 or a location (0, 4) for a sample1972. These two samples are used purely by way of example and the sametechnique applies to each of the samples 1900 to be predicted.

The sample positions of the samples 1970, 1972 to be predicted aremapped according to the direction 1950 associated with the currentprediction mode to respective locations or reference positions 1974,1976 among the reference samples. This mapping may be carried out usingan expression such as that shown below, noting that this is a linearexpression with respect to the coordinate system (x, y):

For horizontal modes 2-17 in the notation of FIG. 11:predicted value(x,y)={1−f(p)}×ref[y+i(p)]+f(p)×ref[y+i(p)+1] withp=A×(x+1)For vertical modes 18-34 in the notation of FIG. 11:predicted value(x,y)={1−f(p)}×ref[x+i(p)]+f(p)×ref[x+i(p)+1] withp=A×(y+1)and where i(p)=floor(p), is the value p rounded down (towards negativeinfinity) to the nearest integer, f(p)=p-i(p) represents the fractionalpart of the value p.

A is an angle parameter indicating the angle of the current mode. Toillustrate, for example, for a horizontal or vertical line, A would be0; for a 45° diagonal line, A would be ±1.

Those skilled in the art would appreciate that integer approximationscan be used to simplify the linear equations, for example, representingthe angle parameter A as a fractional fixed-precision number. In HEVC,the angles have an accuracy of 5 fractional bits.

So, for example, each sample to be predicted is associated with acoordinate position within the current region; and the intra-imagepredictor is configured to detect the reference position for a givensample to be predicted as a function of the coordinate position of thegiven sample to be predicted, the function depending upon the selectedprediction mode.

In example arrangements, the reference position 1974, 1976 is detectedto an accuracy or resolution of less than one sample, which is to saywith reference to the reference sample locations (−5, −1) . . . (15,−1), a fractional value is used for the x coordinate of the referenceposition within the projected set of reference samples 1920. Forexample, the reference position could be detected to a resolution of1/32 of a sample separation, so that the x coordinate of the referencepositions 1974, 1976 is identified to that resolution. The y coordinateof the reference position is −1 in each case, but this is in factirrelevant to the calculations that then take place, which relate tointerpolation along the x axis of the reference samples 1920.

The prediction of the predicted values 1970, 1972 is an interpolation ofthe value applicable to the detected x coordinate of the referencesample position 1974, 1976, for example as described above in theformulae shown earlier.

A similar arrangement is shown schematically in FIG. 18, except that agenerally horizontal prediction mode, for example a prediction modewhich is more horizontal than vertical, such as (for example) mode 14 ofthe set shown in FIG. 11 (or a corresponding number for a similar modein the set shown in FIG. 12) having a prediction direction 2000 is used.The selection of the particular prediction mode applies to the whole ofthe block or region 2010 of samples 2020 to be predicted and theparticular example here is chosen purely for the purposes of the presentexplanation.

In the case of a generally horizontal mode, as discussed above, theprojection circuitry shown in FIG. 16 projects those reference samplesfrom above the block or region 2010 to form an extension 2030 ofreference samples to the left of the region. Once again, the derivationof two example samples to be predicted, samples 2032, 2034, is shown,such that the sample position 2032, 2034 are mapped using the direction2000 into reference positions 2036, 2038 amongst the set of referencesamples 2040. Once again, a similar (x, y) coordinate system is used andthe reference positions 2036, 2038 are expressed to a 1/32 sampleresolution in the y-direction. The x coordinate of the reference samplepositions is −1 but this is irrelevant to the process which follows. Thesample values of the samples to be predicted are obtained in the mannerdescribed above.

In these arrangements, the intra predictor 530 provides an example of adetector configured to detect the reference position as an arrayposition, with respect to an array of the reference samples, pointed toby the prediction direction applicable to the current sample to bepredicted; and a filter configured to generate the predicted sample byinterpolation of the array of reference samples at the detected arrayposition. The detector may be configured to detect the array position toan accuracy of less than one sample such as 1/32 sample.

The intra mode selector 520 the selector may be configured to perform atleast a partial encoding to select the prediction mode.

FIG. 19 schematically illustrates a prediction process.

In the arrangements of FIGS. 17 and 18, for example, the referencesamples 1920, 2440 comprised a single row and column of samples aroundthe current region or block to be predicted. In FIGS. 17 and 18, thissingle row and single column were projected to form either an elongatesingle row in FIG. 17 or an elongate single column in FIG. 18. But theorigin of the reference sample in both cases was a single row and columnto the left of and above the current region.

Further possibilities will now be discussed in which, in at least someexample circumstances, multiple rows and/or multiple columns ofreference samples are used.

FIG. 19 schematically illustrates a situation relating to an 8×8 block2050 of reference samples 2055. The example used here is of an 8×8block, but it will be appreciated that the present techniques can applyto various sizes and indeed shapes of blocks. So, the present techniquescould apply to other sizes such as 4×4, 16×16, 32×32, 64×64 blocks orthe like, or to non-square blocks such as (purely by way of example)8×16 or the like. So, references to the 8×8 blocks are purely for thepurposes of illustrative discussion.

In FIG. 19, two rows of reference samples are used above the block orregion 2050 and two columns of reference samples are used to the left ofthe block or region 2050. Purely by way of example, a predictiondirection 2060 is assumed to have been selected for the block 2050. Thiscould correspond, for example, to the mode 2 in the notation of FIG. 11or a corresponding mode in the notation of FIG. 12. The interpolation orprediction of a particular example predicted sample 2065 will bediscussed, but similar techniques apply to each of the samples 2055 tobe predicted in the block or region 2050.

Discussing first the reference samples, it will be seen that thereference samples in use in FIG. 19 comprise a row and column 2070spatially nearest to the block 2050, along with a further row or column2075 next-adjacent to the row and column 2070. It can also be seen thatthe row and column 2075 extends further (to reference samples 2080,2085) than the row and column 2070, in order to allow for predictionover the range of angles corresponding to the prediction modes 2 . . .34 in FIG. 11 or the equivalent in FIG. 12 to be used. The referencesamples 2080, 2085 can simply be drawn from previously decoded referencesamples in the normal way. If they are unavailable (because they havenot yet been decoded) then next-adjacent samples 2081, 2086 can berepeated to form the samples 2080, 2085 respectively, or alternativelyan extrapolation process can be used as discussed below.

Turning to the interpolation of the sample 2065, it can be seen thatapplying the direction 2060 defined by the current prediction modepoints to a reference position 2090 in the first row and column 2070 ofreference samples. Extending the prediction direction points to afurther position 2095 in the second row and column 2075. The referencesamples around these two reference positions have been annotated asreference samples a . . . g for clarity of the following explanation. Itis also assumed, by way of example, that a 3-tap interpolation processsuch as the process discussed above is used to derive a predictedsample. Of course, other interpolation techniques could be used and thefollowing discussion would simply be adapted accordingly.

FIGS. 20-22 relate to various possible techniques which can be appliedby the intra predictor 530 for making use of two rows and columns ofreference samples in the form shown in FIG. 19.

Considering first, FIG. 20, the reference position 2090 is taken intoaccount and the three samples in the row and column 2070 (namely thereference samples b, d, f) and the reference samples in the row andcolumn 2075 (namely the reference samples c, e, g), which is to say thereference samples within a range 2120 pointed to by the predictiondirection in use, are combined. So in this example, pairs of referencesamples, one from each of the rows/columns 2070, 2075, are combined inrespective groups and the resulting combined reference samples are thenused in an interpolation process. The selection of the reference samplesto be combined is based upon the reference position 2090 in therow/column 2070 and separately on the reference position 2095 in therow/column 2075. This means that a range 2100 of reference samples inthe row/column 2070 is used, and (according to the prediction directionin use) a different—or at least potentially different—range 2120 ofreference samples (c, e, g) is used in dependence upon the referenceposition 2095 in the row/column 2075. The combination takes placebetween the pairs of reference samples, which is to say that referencesamples c and b are combined to form a reference sample h; referencesamples e and d are combined to form a reference sample i; and referencesamples g and f are combined to form a reference sample j. The referencesample h, i and j are then processed by (in this example) a three-tapinterpolation process to form a predicted sample p.

The combination applied to the pairs of reference samples (c, b), (e,d), (g, f) is shown by an arbitrary symbol “Θ” to indicate that variouspossibilities exist for the nature of this combination. This combinationcould be a simple arithmetic mean. In other examples, it could be aweighted mean, for example so as to apply a greater weight to thereference samples (b, d, fin this example) spatially closer to the block2050 than the reference samples (c, e, g) spatially further away fromthe block 2050. For example, in the situation of FIG. 19 in which tworows and columns of reference samples are used, the weighting could be0.6 for the closer reference samples and 0.4 for the further-away samplein each pair, so that (for example) h=0.4c+0.6b. In a situation such asone to be discussed below in which (for example) three or four rows andcolumns of reference samples are used, a weighting could be applied in asimilar manner as follows (where Rn is a reference sample in row/columnn, where n=1 for the row/column spatially closest to the block or regionto be predicted):

Three rows/columns:combined reference sample=0.5R1+0.3R2+0.2R3

Four rows/columns:combined reference sample=0.5R1+0.25R2+0.15R3+0.1R4

Of course, other combinations, or indeed equal combinations, could beused.

So, in the example above, a combination process such as an arithmeticmean or a weighted arithmetic mean is used to combine reference samplesin the rows/columns 2070, 2075 and then the predicted sample generationprocess such as a three-tap interpolation process is used on thecombined values.

As discussed below in connection with FIG. 36, this combination can bedone “in advance” so that a first stage of operation of the intrapredictor 530 can be to combine the multiple rows and columns ofreference samples according to the currently selected predictiondirection, so that that prediction sample generation process proceedswith respect to the combined values as though they were the referencesamples themselves, providing an example in which in which theintra-image predictor is configured to combine the two or more parallellinear arrays of reference samples to form a linear array of referencesamples.

Therefore, FIG. 20 provides an example in which the intra-imagepredictor is configured to combine two or more sets of reference samples(such as (a, c, e) and (b, d, f) in FIG. 20, or (c, e, g) and (b, d, f)in FIG. 20) to derive intermediate reference sample values (h, i, j),and to derive the predicted sample p from the intermediate referencesample values. In example arrangements, the intra-image predictor isconfigured to derive the predicted samples by interpolating amongst theintermediate reference samples. For example, each set of referencesamples may comprises samples from a respective one or the two or moreparallel arrays 2070, 2075 of reference samples. In the case of the useof the samples (c, e, g) in FIG. 20, based around the reference position2095, this is an example in which each set of reference samplescomprises a set, in the respective array of reference samples, pointedto by the prediction direction. In some examples, which the intra-imagepredictor is configured to combine the reference sample values accordingto a weighted combination, in which a weighting applied to a referencesample value decreases with increasing separation of the set ofreference samples containing that reference sample value, from thecurrent region or the current sample to be predicted. For example, theweighting of 0.6 can be used for the reference samples (b, d, f) fromthe array 2070, and the weighting 0.4 can be used for the referencesamples (a, c, e) or (c, e, g) from the array 2075.

In alternative examples, rather than mixing c,b->h a weighted mix(interpolation) of two or more of {a,c,e,g} can be used such that theinterpolated value is spatially aligned with b according to theprediction direction 2060, Then h can be a 50:50 or other weighted mixbetween b and interp (two or more of {a,c,e,g}).

Effectively this involves interpolating the whole column 2075 such thatit is aligned with 2070 according to the direction 2060. Theinterpolated column can then be mixed (by 50:50, 25:75 or anotherweighting) with the column 2070.

During the interpolation process described above, since the projectionof the column 2075 to be spatially aligned with samples of the column2070 according to the current prediction direction will requireinterpolation, supersampling (so as to generate interpolated samples ata smaller spatial resolution than the original reference samples) couldbe used to reduce any negative impact of the interpolation process(since interpolation can in some situations soften data or reduces highfrequency detail).

Another option is to use so-called non-uniform sampling, to combine thetwo columns into a supersampled data set. The phasing of the tworegularly set of sampled values is determined by the angle of thecurrently selected prediction direction. To avoid effects of noise, thenew reference samples may be low-pass filtered, either in a subsequentprocess, or as part of the supersampling process.

In another example method of operation, each row/column 2070, 2075 isused individually to generate an intermediate predicted sample value,and the intermediate predicted sample values are then combined.

Therefore these arrangements provide examples in which each set ofreference samples comprises a set, in the respective array of referencesamples, or of values interpolated from the respective array ofreference samples, pointed to by the prediction direction.

Looking first at FIG. 21, this relates to the use of the range 2100 ineach row/column being aligned only with the reference position 2090 inthe row/column 2070, so that the reference samples a, c and e arecombined (for example, by the three-tap interpolation process) toproduce a first intermediate predicted sample p1. The reference samplesb, d and f in the row/column 2070 are combined by a similar process toproduce a second intermediate predicted value p2. The values p1 and p2can then be combined, for example, by an arithmetic mean or a weightedmean (for example, as discussed above, placing a greater weight such as0.6 on the intermediate predicted sample value p2 and a reduced weightsuch as 0.4 on the intermediate predicted sample value p1, given that itis generated from reference samples further away from the block 2050) togenerate the final predicted sample value p2200.

A similar arrangement is shown in FIG. 22, but making use of the rangeof reference samples 2100 in the row/column 2070 and the range 2120 inthe row/column 2075, which is to say that reference samples in therow/column 2075 around the reference position 2095 in that row/columnare used.

So, the first intermediate predicted sample value p1 is generated fromthe reference samples c, e and g and the second intermediate predictedsample value p2 is generated from the reference samples b, d and f. Asbefore, these can be combined by any of the processes discussed above toform the final predicted sample value p.

The examples discussed with reference to FIGS. 20-23 relate to a pair ofrows/column 2070, 2075. If more than two rows/columns are use, theneither the processes discussed above could be applied individually. So,in the case of FIG. 20, for n rows/columns, where n is at least two, allof the reference samples within respective ranges 2100, 2120 and thelike of each individual row/column are combined to form a set of threeintermediate reference samples h, i, j which are then combined. In thecase of FIGS. 21 and 22, for n rows/columns, where n is at least two, nintermediate predicted sample values are generated and are thencombined, for example using a weighted combination.

In example arrangements, the controller 343 can vary the weightingaccording to one or more properties or parameters of the currentinterpolation process. For example, such a parameter may be the blocksize of the current block to be interpolated. The weighting could bedetected by the controller 343 from a predetermined or programmable (forexample via parameter sets communicated as part of the compressed datastream) set of weight values, or derived using a predetermined orprogrammable function. An example of such a relationship (whetherdefined by a look-up or a function) is:

block size up to a threshold block size (such as a threshold of 4×4, 8×8or (in the case of non-square blocks) one dimension being up to 8samples, or one size being up to 4 samples):weighting is 25:75 (25% forthe further row/column of reference samples or interpolated samplesderived from them and 75% for the closer row/column); or

block size greater than the threshold block size:weighting is 50:50

In other examples, the (or a) parameter may represent a spatialseparation, in sample rows or columns or along the prediction direction,of the current sample to be interpolated from the nearest row/column ofreference samples. In the example of FIG. 19, the sample position 2065is in the fourth column of samples to be interpolated, starting at thereference column 2070. A mapping can be used between weightings andcolumn separation (or row separation in the case of a generally verticalprediction direction, such as: m=weighting applied to nearer column/rowof reference samples or to interpolated samples derived from them;

n=weighting applied to farther column/row of reference samples, or tointerpolated samples derived from them;

s=separation of current sample position from nearest reference incolumns/rows

bs=block size in that dimension (in columns or rows, whichever is usedto define s)

For example:

m:n=s: (s+1)

Or:

m=0.25+(0.25*s/bs); n=(1−m)

The weighting used can be generated by applying two or more of thesefunctions as discussed, for example with m,n being respective productsof a weighting m,n derived by block size and a weighting m,n derived bysample position.

In other words, the influence or contribution of a non-adjacent row orcolumn of reference samples increases as the separation of the sampleposition to be predicted from that row/column increases. For example,for samples to be predicted which are adjacent to the nearest row/columnof reference samples, the influence of another (further away) row/columnof reference samples may be expected to be lower than if the sample tobe predicted is a long way (say, 8 pixels or more) from the row/columnof reference samples adjacent to the current block, then the influenceof the non-adjacent (such as next) row/column of reference samples maybe expected to be larger.

Therefore, in examples, the intra-image predictor is configured tocombine the intermediate sample values according to a weightedcombination, in which a weighting applied to an intermediate samplevalue derived from reference samples non-adjacent to the current imageregion increases with increasing separation of the set of referencesamples, from which that intermediate sample value as generated, fromthe current sample to be predicted.

In examples, the intra-image predictor is configured to combine thereference sample values according to a weighted combination, in which aweighting applied to a reference sample value non-adjacent to thecurrent image region increases with increasing separation of the set ofreference samples containing that reference sample value, from thecurrent sample to be predicted.

Various different options of these combinations can be tested as trialencodings and one selected, for example according to a lowest sum ofabsolute differences (SAD) amongst those tested, for use in encoding,with the selection being indicated by parameter data communicated aspart of the compressed data stream.

Alternatively reference samples in sub-groups in rows/columns could becombined using the techniques of FIG. 20, to form sub-combinations whichcan then be processed using the techniques shown in FIGS. 21 and 22. Anexample of this arrangement is given below for an example arrangement offour rows/columns of reference samples, numbered 1-4, where row/column 1is spatially closest to the current block or region:

Rows/columns 1 & 2:

-   -   generate first combined reference samples as in FIG. 20    -   generate a first intermediate predicted sample value from the        first combined reference sample values

Rows/columns 3 & 4:

-   -   generate second combined reference samples as in FIG. 20    -   generate a second intermediate predicted sample value from the        second combined reference sample values

Then:

-   -   generate a final predicted sample value p from the first and        second intermediate predicted sample values.

Various options will now be discussed relating to the number of rows andcolumns of reference samples. Again, as before, the examples arediscussed with relation to an 8×8 block 2400 of samples to be predicted,but the same techniques are applicable to other sizes and/or shapes ofblocks.

Note however that in some examples, certain block sizes and/or shapesmay be excluded or restricted in their application of the presenttechniques, for example small blocks, such as blocks having eitherdimension equal to four samples or fewer. Examples of such arrangements,and alternatives applicable to these techniques, will be discussedfurther with reference to FIGS. 41 to 46 below.

Also, in the operation of an intra mode selector as discussed below,certain directional modes may be excluded from the present techniques.

Therefore, FIGS. 21 and 22 provide examples in which the intra-imagepredictor is configured to derive the predicted samples by interpolatingamongst one or more sets of reference samples. For example, theintra-image predictor can be configured to interpolate amongst two ormore sets of reference samples (such as (a, c, e) and (b, d, f) in FIG.21, or (c, e, g) and (b, d, f) in FIG. 22) to derive a respectiveintermediate sample value p1, p2 from each set of reference samples, andto combine the intermediate sample values to derive the predicted samplep. In example arrangements set of reference samples comprises samplesfrom a respective one or the two or more parallel arrays 2070, 2075 ofreference samples. In the example of FIG. 22, based around the referencepositions 2090, 2095, each set of reference samples comprises a set, inthe respective array of reference samples, pointed to by the predictiondirection. As discussed above, the intra-image predictor 530 can beconfigured to combine the intermediate sample values according to aweighted combination, in which a weighting applied to an intermediatesample value decreases with increasing separation of the set ofreference samples, from which that intermediate sample value asgenerated, from the current region (so that in the example given above,a weighting of 0.6 is applied to the closer array 2070 and a weightingof 0.4 is applied to the further array 2075) or the current sample to bepredicted.

FIG. 23 schematically illustrates a single row/column 2410 of referencesamples. If this was the only option available to the intra predictor,the operation would correspond to a previously proposed intra predictor,but the use of a single row/column 2410 could be provided in an intrapredictor forming an embodiment of the present disclosure in the contextof its optional selection (by the intra mode selector) in combinationwith the optional selection (by the intra mode selector) of one or moreother techniques shown in FIGS. 24 to 26.

FIG. 24 shows a pair of rows/columns 2500, 2510 in which, as discussedabove, the row/column 2510 is extended by one or more samples 2520, 2530so as to allow for the use of the full range of prediction directionsdiscussed above.

Similarly, FIG. 25 schematically represents three rows/columns ofreference samples 2600, 2610, 2620. The row/column 2620 is extended byone or more samples 2630, 2640 with respect to the second row/column2610, for the same reasons.

Finally as an example, although not representing a limit on the numberof rows/columns which can be used, FIG. 23 schematically illustratesfour rows/columns of reference samples, mainly rows/columns 2700, 2710,2720, 2730. Once again, the row/column 2730 is extended by one or morereference samples 2740, 2750 with respect to the third row/column 2720,for the same reasons as those discussed above.

FIGS. 27-30 show, for the example cases of FIGS. 23-26 respectively, aprojected version of the multiple rows/columns of reference samples(shown as shaded blocks in FIGS. 27-30). Note that in at least someexamples, the projection process can be dependent upon the predictiondirection in use, so a single example for an arbitrary predictiondirection is employed in FIGS. 27-30. With respect to the projectedreference sample, a similar technique employed to that in FIG. 17 can beused to derive reference positions and reference samples to apply thetechnique of FIG. 19.

FIG. 31 schematically represents at least part of an operation of anintra mode selector such as the intra mode selector 520 of FIG. 8described above.

The intra mode selector can operate to detect an appropriate mode foruse in intra prediction of a current block or region with respect to aset of reference samples 3205. Various techniques have been proposed forachieving this, such as (at least partial) trial encoding and/oranalysis of properties of the reference samples, to select theprediction operation amongst the candidate prediction operations.

In the present examples, any of these techniques can be used, with (insome example arrangements) the techniques being repeated, or applied inmultiple instances, in respect of multiple permutations of the number ofrows/columns of reference samples.

Here, the term “permutation” is used to indicate a group of rows/columnsof reference samples. The group might include a row/column spatiallynearest to the current block or region, and zero or more next-adjacentrows/columns each progressively spatially further away from the currentblock or region.

The term “prediction operation” can be used to describe a mode ordirection and/or an associated permutation of rows/columns. Therefore,the intra-mode selector can, in example embodiments, be configured toselect, from a set of candidate prediction operations each defining atleast a prediction direction, a prediction operation for prediction ofsamples of a current region of a current image, the current regioncomprising an array of two or more rows and two or more columns ofsamples. The intra-image predictor is configured to derive predictedsamples of the current region with respect to one or more of a group ofreference samples of the same image in dependence upon a predictiondirection, defined by the selected prediction operation, between acurrent sample to be predicted and a reference position amongst thereference samples. For at least some of the candidate predictionoperations, the group of reference samples comprises two or moreparallel linear arrays (such as rows, columns, rows and columns (notingthat a row and column are still a linear array even with a “corner”),and/or projected arrays) of reference samples disposed at differentrespective separations from the current region.

The intra mode selector comprises a mode properties detector 3200 whichacts to detect the encoding properties of each mode under test (and, inat least some examples, of each mode with each permutation Pn of numbersn of rows/columns of reference samples available with that mode, where nranges from 1 to a maximum limit of at least two). A coding efficiencydetector 3210 detects the coding efficiency for each mode/permutationtested by the mode properties detector 3200. The coding efficiency canrelate to aspects such as the amount of data needed to encode theresidual image which would arise where that mode and permutation ofrows/columns to be used, along with the amount of data needed to signalthe use of that mode/permutation.

In the selection of a prediction direction, example arrangements testall 34 directional modes with a simple SAD (sum of absolute differences)test to derive a shortlist of modes most likely to be useful for thecurrent block. The shortlist of modes is then tested with a full encodeto select a prediction mode for use.

The current techniques using multiple rows and/or columns of referencesamples my in some situations be slower (or have higher processingrequirements) than using a single row/column. To alleviate this at theencoder side where such a decision is made and then communicated to thedecoder, example arrangements may be arranged to use the multirow/column prediction only for testing the shortlist of modes.

Based on the detection by the coding efficiency detector 3210, a modeselector 3220 selects a prediction mode 3230 to be sent to the intrapredictor 530, and information 3240 is also sent to a mode encoder 3245such as a part of the controller 343 which encodes data 3250 formingpart of the encoded data stream indicating the mode/permutation in use,which is to say, data identifying the prediction operation selected foreach region of the image.

Therefore, this provides an example in which the intra mode selector isconfigured to select amongst two or more groups of reference samples,each group comprising a respective different number of parallel arraysof reference samples.

FIGS. 32 and 33 are respective schematic flow charts representingpossible methods of operation of the intra mode selector of FIG. 31.

In FIG. 32, steps 3300 . . . 3310, carried out in as many instances(whether in series, as schematically illustrated here, or in parallel)as there are available permutations P of rows/columns of referencesamples, relate to the mode properties detector 3200 and codingefficiency detector 3210 detecting a best mode using each possiblepermutation of rows/columns of reference samples. So, a best mode may beselected using a single row/column of reference samples, another bestmode may be selected using two rows/columns of reference samples and soon. Then, at a step 3320, the coding efficiency detector 3210 detectsthe efficiencies of each of the modes and permutations detected at thestep 3300 . . . 3310 so that at a step 3330 the mode selector 3220selects a single mode and permutation of rows/columns and the modeencoder 3245 encodes information defining the selected mode andpermutation.

In an alternative mode of operation in FIG. 33, an initial stage duringthe encoding process is to establish use of a particular permutation ofn rows/columns (where n is at least two, so the permutation might be tworows/columns) and communicate this to the decoder in, for example, aparameter set, at a step 3400. Then, at a step 3410, the mode propertiesdetector 3200, the coding efficiency detector 3210 and the mode selector3220 cooperate to select a best prediction mode using the particularestablished permutation of rows/columns and at a step 3420 the modeencoder 3425 encodes information defining that mode.

At the decoder side, an intra mode selector is shown by way of examplein FIG. 34, comprising an encoded data detector 3500 which detects datasuch as the data 3250 in the encoded data stream defining a particularmode (and optionally a set of rows/columns) to be used, and a modeselector 3510 provides information 3520 to the intra predictor 530indicating which mode and permutation of rows/columns to use, that is tosay, it is configured to detect encoded data identifying the predictionoperation selected for each region of the image.

FIGS. 35 and 36 are respective flow charts schematically representingmethods of operation of an intra predictor such as the intra predictor530 of FIG. 8.

FIG. 35 relates primarily to the operations shown in FIGS. 21 and 22above, in which, at a step 3600, an intermediate predicted sample valuesuch as the intermediate predicted sample value p2 is interpolated froma first set of reference samples, for example the set in a closestrow/column of reference samples to the current block or region, andthen, using as many iterations as there are rows/columns of referencesamples, a step 3610 represents the interpolation of an intermediatepredicted sample value p1 from each remaining set of reference samplesfrom a respective row/column of reference samples. As discussed above,for those rows/columns of reference samples further away from thecurrent block or region than the closest row/column of reference sample,the selection of which reference samples to use could be, for example,by detecting a reference position in each row/column according to theprediction direction or by using corresponding locations to thoseidentified in the closest row/column. Then, at a step 3620, theintermediate predicted sample values are combined, for example by aweighted mean, to generate the final predicted sample value.

FIG. 36 relates more to the process shown in FIG. 20, in which at a step3700, respective members of the two or more sets of reference samples(with sets other than the set in the closest row/column being definedfor example by tracking a further reference position along theprediction direction) into a combined set (h, i, j of FIG. 20), and theninterpolating a predicted sample value p from the combined set a step3710.

As an optional feature of the operation of the intra predictor, FIG. 37schematically illustrates a technique for providing the additionalreference samples 2520, 2530, 2630, 2640, 2740, 2750, or at least someof them, without necessarily having to simply repeat nearby or adjacentreference samples.

Referring to FIG. 37, a missing reference sample detector 3800 detectsthat a reference sample is not available. This could be because a sampleat that position has not yet necessarily been decoded. In some examples,the sample has not been decoded. In other examples, a sample could havebeen decoded, but because it lies outside a current coding unit it couldbe that the hardware requirements such as buffering needed to make thatreference sample available for the current prediction would beunreasonably high and so a design decision is taken such that the sampleis not provided for prediction. For example, referring to FIGS. 25 and26, these considerations may indicate that the samples 2530, 2640, 2642are not available if they are outside the current coding unit. Toaddress these issues, in example arrangements an extrapolator 3810generates the required value 3820 using an extrapolation process basedon multiple ones of the available reference samples. Operation of thisarrangement is shown by schematic flowchart of FIG. 38 in which a step3900 relates to the detection by the detector 3800 of the missingreference samples and a step 3910 relates to the extrapolation by theextrapolator 3810 of the required values.

FIG. 39 is a schematic flowchart illustrating an image encoding methodcomprising:

selecting (at a step 4000), from a set of candidate predictionoperations each defining at least a prediction direction, a predictionoperation for prediction of samples of a current region of a currentimage, the current region comprising an array of two or more rows andtwo or more columns of samples; and

deriving (at a step 4010) intra-image predicted samples of the currentregion with respect to one or more of a group of reference samples ofthe same image in dependence upon a prediction direction, defined by theselected prediction operation, between a current sample to be predictedand a reference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

FIG. 40 is a schematic flowchart illustrating an image encoding methodcomprising:

selecting (at a step 4100), from a set of candidate predictionoperations each defining at least a prediction direction, a predictionoperation for prediction of samples of a current region of a currentimage, the current region comprising an array of two or more rows andtwo or more columns of samples; and

deriving (at a step 4110) intra-image predicted samples of the currentregion with respect to one or more of a group of reference samples ofthe same image in dependence upon a prediction direction, defined by theselected prediction operation, between a current sample to be predictedand a reference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

Note that the linear arrays do not necessarily have to have the sameextent; they just have to be parallel) and so different numbers of rowsand columns are not excluded by this wording.

In at least some of the example arrangements discussed above, the use ofmultiple rows or columns of reference samples may be constrained toinstances where the “extra” (non-adjacent) rows or the multiple columns(or both) lie in the same coding tree unit as the current block. Theseconditions can be applied individually or together.

As mentioned above, in at least some examples, certain block sizesand/or shapes may be excluded or restricted in their application of thepresent techniques, for example small blocks, such as blocks havingeither dimension equal to four samples or fewer.

As examples of such arrangements, FIG. 41 schematically illustrates theprediction of a block 4100 of 8×8 samples in which two parallel lineararrays 4110 (or in other words, two upper rows and two left columns) ofreference samples are used as references in the prediction of samples ofthe block 4100 according to any of the techniques described above.

In an example of FIG. 42, a 4×4 block 4200 of samples is beingpredicted. Here, only one linear array 4210 (or one upper row and oneleft column) of reference samples is used.

One potential reason for handling the different block sizes in this wayis as follows. In the case of the 4×4 block 4200, if a second row/columnof reference samples were used, then for a given sample to be predicted(an arbitrary sample within the block 4200) the relative separation of(i) reference samples in the second (farther away) row/column from thegiven sample to be predicted; and (ii) reference samples in the first(nearer) row/column from the given sample to be predicted is large. Inother words, because the block 4200 is small, there is a significantfractional difference in distance between any given sample to bepredicted and the extra row(s)/column(s) of reference samples.

In contrast, in the example 8×8 block 4100, for at least some of thesamples to be predicted, the fractional difference in distance betweenany given sample to be predicted and the extra row(s)/column(s) ofreference samples is somewhat smaller than is the case for the 4×4 block4200.

This variation in comparative distance for the given sample to bepredicted in the smaller block 4200 implies that the extrarow(s)/column(s) of reference samples are potentially less relevant toan accurate prediction of the samples of the block 4200, which would inturn imply that it is better not to use the extra row(s)/column(s).

Even if the use of the extra row(s)/column(s) does not bring an actualdisadvantage to the prediction of smaller blocks, another potentialargument for not using the extra row(s)/column(s) is that the use of theextra row(s)/column(s) potentially requires further processing resources(compared to the use of a single row/column) and so in the case of smallblocks such as 4×4 blocks or smaller, this additional processingresource is not justified by relatively small gains available with thesmaller blocks.

So, in some examples, additional row(s)/column(s) of reference samples(beyond one, or alternatively beyond another upper limit value such as2) are excluded for any square or non-square block having at least onedimension of fewer than a threshold size such as 8 samples. Theexclusion is applied at, for example, the mode properties detector 3200of FIG. 32 and/or in the steps 3300 . . . 3310 or 3400 of FIGS. 32 and33.

It will be appreciated that a threshold other than 8 samples could beused. For example, 16 samples could be used as the threshold value.

Other example arrangements, however, can allow the partial or restricteduse of these techniques in the case of non-square blocks having onedimension of fewer than a threshold size such as 8 samples and anotherdimension which is not smaller than the threshold size. Examples of suchnon-square blocks (for luminance and/or chrominance) might include an8×4 block (in other words, in a landscape orientation, wider than itsheight) or a 4×8 block (in other words, in a portrait orientation,taller than its width). Other block sizes are possible in which theseconditions are met (such as 4×16, 2×8, 16×4, 8×2 blocks or the like). Insome example systems, luminance blocks have a minimum dimension of 4samples but chrominance blocks (for example in a sub-sampled chrominanceformat such as 4:2:0) can have a dimension of 2 samples.

Note that a comparison of whether a value p is at least a threshold Thr1can be expressed equivalently as:

p>=Thr1; or

p>Thr2 (where Thr2=Thr1-1)

Similarly, a test as to whether p is smaller than Thr1 can be expressedas:

p<Thr1; or

p<=Thr2

Therefore the discussions provided here as to (for example) whether ablock dimension is “at least a threshold size” or is “smaller than thethreshold size” where for example the threshold is 8 samples, should beread as indicating each of the respective entirely equivalentformulations just discussed.

FIGS. 43 and 44 schematically represent respective examples of an 8×4block 4300 of samples to be predicted, and a 4×8 block 4400 of samplesto be predicted.

Considering first the block 4300, the comparative distance as betweenfirst (nearest to the current region) and subsequent (further away)arrays of reference samples as discussed above can be consideredseparately in a vertical sense (in which the comparative distance for anarbitrary given sample to be predicted is relatively large, for example5:4 for a given sample in a fourth row, or 2:1 for a sample in the toprow) and in a horizontal sense (in which the comparative distance for anarbitrary given sample to be predicted is relatively smaller, at leastfor some of the samples, for example, 9:8 for a sample in the 8^(th)column). Therefore, it can be considered that the potential advantagesof the use of multiple row(s)/column(s) of reference samples may applyin respect of the left column of reference samples but may potentiallynot apply in respect of the upper row of reference samples. Conversely,if it were to be considered that the use of multiple row(s)/column(s) ofreference samples can be disadvantageous in respect of smaller blocks,then it may be considered that the potential disadvantages apply inrespect of the upper row of reference samples but potentially not in thecase of the left column of reference samples.

Therefore, in the example of FIG. 43, the use of multiplerow(s)/column(s) of reference samples is allowed for the left column4310 of reference samples but excluded for the upper row 4320 ofreference samples.

Note that this arrangement does not necessarily imply that a particularnon-square block has to be encoded using reference samples in the mannerof FIG. 43, but rather that such an arrangement is allowed as an option,to be selected from as a candidate by the permutation and mode selectionarrangements discussed above. In alternative embodiments, of course,such an arrangement could be handled as a compulsory way of handling anon-square block with one dimension below the threshold size.

The arrangement of FIG. 43 can be particularly useful where it isdesired (in the design or implementation of an encoding and/or decodingapparatus) to avoid the use of more than one line store in order toprovide reference samples from a block or unit above the current regionin the image, or in other words to store one or more lines or rows ofpreviously encoded and decoded samples from a coding unit directly abovea current coding unit. Here, it is noted that line stores are consideredto be relatively “expensive”, the term implying that they can require asignificant amount of memory storage for their implementation, which canin turn require a significant amount of logic circuitry in, for example,an integrated circuit implementation of an encoder and/or decoder. Theprovision of line stores therefore represents a compromise between thepotentially heavy requirements in terms of logic circuitry and thepotential improvements in coding efficiency provided by the presenttechniques in which multiple rows and/or columns of reference samplesmay be used. Such reference samples may be readily available within aparticular coding unit, but access to reference samples from outside thecurrent coding unit can potentially be restricted by the implementation(or lack of implementation) of such line stores. In such circumstancesthe arrangement of FIG. 43 (at least at the top of a coding unit, butoptionally at other image locations) can avoid the need for more thanone line store but still provide the potential advantages of multiplelinear arrays in the column direction. Therefore, in examples, when n isless than m, the set of candidate prediction operations comprisesprediction operations in which the number of rows of reference samplesis one and the number of columns of reference samples is greater thanthe number of rows of reference samples.

In FIG. 44, a 4×8 block of samples to be predicted is shown. Here, forthe same reasons as discussed in connection with FIG. 43, more than onerow 4410 of upper reference samples are provided (or allowed) but onlyone left column 4420 of reference samples is provided or allowed.

Note that in the examples of FIGS. 43 and 44, two columns 4310 and tworows 4410 are shown. In some example embodiments, this could be an upperlimit on the number of linear arrays allowed in that direction. In otherexamples, multiple rows/columns of reference samples could be allowedfor selection (as part of the mode/permutation selection arrangementsdiscussed above) as reference samples extending in the longer dimensionof the non-square block (having a shorter dimension smaller than thethreshold size).

Note that instead of limiting the use of multiple rows/columns so thatonly one row/column is allowed for reference samples extending in theshorter dimension of the non-square block, alterative arrangements couldimpose a different maximum number (greater than one) of rows/columns inthat direction, such as a maximum of two rows/columns. The maximumnumber could be different for rows of reference samples (for landscapeorientation blocks) and for columns of reference samples (for portraitorientation blocks). Or the maximum numbers could be the same.

Therefore, in FIGS. 43 and 44, the linear arrays of reference samplescomprise one or more rows and one or more columns of reference samplesdisposed with respect to the current region; and for a non-squarecurrent region of n rows and m columns of samples, where n does notequal m, for at least some of the candidate prediction operations, thegroup of reference samples comprises different respective numbers ofrows and columns of reference samples. A limit can be imposed in respectof the number of rows or columns as discussed above. For example, when nis less than a threshold value but m is at least the threshold value,the set of candidate prediction operations comprises predictionoperations in which no more than a first upper limit of rows ofreference are used (and in at least examples does not comprise candidateprediction operations not complying with this condition); and when m isless than a threshold value but n is at least the threshold value, theset of candidate prediction operations comprises prediction operationsin which no more than a second upper limit of columns of reference areused (and in at least examples does not comprise candidate predictionoperations not complying with this condition). The first upper limit maybe different to the second upper limit but in some examples they areequal. For example, the first upper limit and the second upper limit maybe equal to one, but they could be different (to one another) and/or oneor both could be values other than one, such as two. In some examples,the threshold value is eight samples.

FIG. 45 schematically represents a potential modification of the intramode selector shown in FIG. 31 and described above. Features which aresubstantially identical to those described in connection with FIG. 31will not be described in detail again.

As before, the term “permutation” is used to indicate a group ofrows/columns of reference samples. The group might include a row/columnspatially nearest to the current block or region, and zero or morenext-adjacent rows/columns each progressively spatially further awayfrom the current block or region. The term “prediction operation” can beused to describe a mode or direction and/or an associated permutation ofrows/columns.

A controller 4520 is responsive to an indication 4510 of the block sizein use and controls the operation of the mode properties detector 3200′so as to detect the coding efficiencies of permutations excluding theuse of multiple rows/columns of reference samples extending in theshorter dimension of a block having one dimension smaller than thethreshold size such as 8 samples. The mode properties detector 3200′acts as described above to detect the encoding properties of each modeunder test (and of each mode with each permutation Pn of numbers n ofrows/columns of reference samples available with that mode and blocksize, where n ranges from 1 to a maximum limit of at least two). Asabove, the coding efficiency detector 3210 detects the coding efficiencyfor each mode/permutation tested by the mode properties detector 3200and the process continues as discussed above, but based on thepermutations allowed by the controller 4520.

In some other examples, the block size and/or shape can itself beselected, at least in part, by the apparatus of FIG. 45 as part of itsselection process, in that a variety of block sizes, shapes, predictionmodes and permutations of numbers or rows/columns of reference samplescan be tested and assessed by the coding efficiency detector, with themode selector 3220 choosing the best, or most efficient, or leastoverall cost option (where cost is a measure which may be dependent atleast in part upon overall resulting bitrate). In such an arrangement,the controller 4520 can control the operation so as to allow testing ofnon-square blocks, for example non-square blocks where one dimension issmaller than the threshold, using numbers of rows/columns of referencesamples limited to or including the arrangements described above withreference to FIGS. 43 and 44.

FIG. 46 schematically illustrates a modification of the flowchart ofFIG. 32 or the flowchart of FIG. 33 discussed above, and in particularillustrates (purely as an example) two steps which can precede the steps3300 or 3400. At a step 4600, the current block size is detected and ata step 4610 the set of available permutations of numbers of rows/columnsof reference samples to be handled by the remainder of the process ismodified (if appropriate to the block size in use) according to theprinciples discussed above.

The examples given above relate to so-called small blocks, in which one(but not both) of the numbers of rows and columns is less than thethreshold value. However, similar principles can be applied tonon-square blocks more generally. An example of this being useful mightoccur in the context of (say) a 32×32 block being split for predictioncoding into two 32×16 landscape format blocks. Each of these exceeds athreshold dimension of 8 but given that a reason for the splitting mightbe that there is an image content discontinuity in the verticaldirection which is not well predicted in the 32×32 blocks, it may proveuseful to provide a greater number of columns than rows of referencesamples in this situation.

As general examples, in FIG. 47, a landscape orientation (wider thantall) block of n rows and m columns of samples, where n does not equalm, is shown in schematic form. The linear arrays of reference samplescomprise one or more rows 4710 and one or more columns 4720 of referencesamples disposed with respect to the current region; and for thenon-square current region 4700, for at least some of the candidateprediction operations, the group of reference samples comprisesdifferent respective numbers of rows and columns of reference samples.In this example, a number of columns of reference samples is used in atleast some candidate prediction operations which is larger than thenumber of rows of reference samples.

In FIG. 48, a portrait orientation (taller than wide) block of n rowsand m columns of samples, where n does not equal m, is shown inschematic form. The linear arrays of reference samples comprise one ormore rows 4810 and one or more columns 4820 of reference samplesdisposed with respect to the current region; and for the non-squarecurrent region 4800, for at least some of the candidate predictionoperations, the group of reference samples comprises differentrespective numbers of rows and columns of reference samples. In thisexample, a number of rows of reference samples is used in at least somecandidate prediction operations which is larger than the number ofcolumns of reference samples.

As before, the discussion of FIGS. 47 and 48 does not necessarily imply(though it could so imply in some examples) that different numbers ofrows and columns of reference samples are imposed upon a non-squareblock, but rather that such arrangements are available as candidateprediction operations to be selected from in the manner discussed above.

FIGS. 47 and 48 therefore provide examples, in the context of blocks4700, 4800 of n rows by m columns, of arrangements in which:

when n is less than m, the set of candidate prediction operationscomprises prediction operations in which the number of columns ofreference samples is greater than the number of rows of referencesamples; and

when m is less than n, the set of candidate prediction operationscomprises prediction operations in which the number of rows of referencesamples is greater than the number of columns of reference samples.

In so far as embodiments of the disclosure have been described as beingimplemented, at least in part, by software-controlled data processingapparatus, it will be appreciated that a non-transitory machine-readablemedium carrying such software, such as an optical disk, a magnetic disk,semiconductor memory or the like, is also considered to represent anembodiment of the present disclosure. Similarly, a data signalcomprising coded data generated according to the methods discussed above(whether or not embodied on a non-transitory machine-readable medium) isalso considered to represent an embodiment of the present disclosure.

It will be apparent that numerous modifications and variations of thepresent disclosure are possible in light of the above teachings. It istherefore to be understood that within the scope of the appendedclauses, the technology may be practised otherwise than as specificallydescribed herein.

Respective aspects and features are defined by the following numberedclauses:

1. An image encoding apparatus comprising:

a selector configured to select, from a set of candidate predictionoperations each defining at least a prediction direction, a predictionoperation for prediction of samples of a current region of a currentimage, the current region comprising an array of two or more rows andtwo or more columns of samples; and

an intra-image predictor configured to derive predicted samples of thecurrent region with respect to one or more of a group of referencesamples of the same image in dependence upon a prediction direction,defined by the selected prediction operation, between a current sampleto be predicted and a reference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

2. Apparatus according to clause 1, in which the intra-image predictoris configured to derive the predicted samples by interpolating amongstone or more sets of reference samples.

3. Apparatus according to clause 1 or clause 2, in which the intra-imagepredictor is configured to interpolate amongst two or more sets ofreference samples to derive a respective intermediate sample value fromeach set of reference samples, and to combine the intermediate samplevalues to derive the predicted sample.4. Apparatus according to clause 3, in which each set of referencesamples comprises samples from a respective one or the two or moreparallel arrays of reference samples.5. Apparatus according to clause 4, in which each set of referencesamples comprises a set, in the respective array of reference samples,or of values interpolated from the respective array of referencesamples, pointed to by the prediction direction.6. Apparatus according to clause 3, in which the intra-image predictoris configured to combine the intermediate sample values according to aweighted combination, in which a weighting applied to an intermediatesample value decreases with increasing separation of the set ofreference samples, from which that intermediate sample value asgenerated, from the current region.7. Apparatus according to clause 3, in which the intra-image predictoris configured to combine the intermediate sample values according to aweighted combination, in which a weighting applied to an intermediatesample value derived from reference samples non-adjacent to the currentimage region increases with increasing separation of the set ofreference samples, from which that intermediate sample value asgenerated, from the current sample to be predicted.8. Apparatus according to any one of the preceding clauses, in which theintra-image predictor is configured to combine two or more sets ofreference samples to derive intermediate reference sample values, and toderive the predicted sample from the intermediate reference samplevalues.9. Apparatus according to clause 8, in which the intra-image predictoris configured to derive the predicted samples by interpolating amongstthe intermediate reference samples.10. Apparatus according to clause 8, in which each set of referencesamples comprises samples from a respective one or the two or moreparallel arrays of reference samples.11. Apparatus according to clause 10, in which each set of referencesamples comprises a set, in the respective array of reference samples,pointed to by the prediction direction.12. Apparatus according to clause 8, in which the intra-image predictoris configured to combine the reference sample values according to aweighted combination, in which a weighting applied to a reference samplevalue decreases with increasing separation of the set of referencesamples containing that reference sample value, from the current region.13. Apparatus according to clause 8, in which the intra-image predictoris configured to combine the reference sample values according to aweighted combination, in which a weighting applied to a reference samplevalue non-adjacent to the current image region increases with increasingseparation of the set of reference samples containing that referencesample value, from the current sample to be predicted.14. Apparatus according to clause 8, in which the intra-image predictoris configured to combine the two or more parallel linear arrays ofreference samples to form a linear array of reference samples.15. Apparatus according to any one of the preceding clauses, in whichthe selector is configured to select amongst two or more groups ofreference samples, each group comprising a respective different numberof parallel arrays of reference samples.16. Apparatus according to any one of the preceding clauses, in whichthe selector is configured to perform at least a partial encoding toselect the prediction operation amongst the candidate predictionoperations.17. Apparatus according to any one of the preceding clauses, in whichthe controller is configured to encode data identifying the predictionoperation selected for each region of the image.18. Apparatus according to any one of the preceding clauses, in which:

the linear arrays of reference samples comprise one or more rows and oneor more columns of reference samples disposed with respect to thecurrent region; and

for a non-square current region of n rows and m columns of samples,where n does not equal m, for at least some of the candidate predictionoperations, the group of reference samples comprises differentrespective numbers of rows and columns of reference samples.

19. Apparatus according to clause 18, in which:

when n is less than a threshold value but m is at least the thresholdvalue, the set of candidate prediction operations comprises predictionoperations in which no more than a first upper limit of rows ofreference samples are used; and

when m is less than a threshold value but n is at least the thresholdvalue, the set of candidate prediction operations comprises predictionoperations in which no more than a second upper limit of columns ofreference samples are used.

20. Apparatus according to clause 18, in which:

when n is less than m, the set of candidate prediction operationscomprises prediction operations in which the number of columns ofreference samples is greater than the number of rows of referencesamples; and

when m is less than n, the set of candidate prediction operationscomprises prediction operations in which the number of rows of referencesamples is greater than the number of columns of reference samples.

21. Apparatus according to clause 18, in which:

when n is less than m, the set of candidate prediction operationscomprises prediction operations in which the number of rows of referencesamples is one and the number of columns of reference samples is greaterthan the number of rows of reference samples.

22. Apparatus according to clause 19, in which the first upper limit isequal to the second upper limit.

23. Apparatus according to clause 22, in which the first upper limit andthe second upper limit are equal to one.

24. Apparatus according to clause 20, in which the threshold value iseight samples.

25. Video storage, capture, transmission or reception apparatuscomprising apparatus according to any one of the preceding clauses.

26. An image decoding apparatus comprising:

a selector configured to select, from a set of candidate predictionoperations each defining at least a prediction direction, a predictionoperation for prediction of samples of a current region of a currentimage, the current region comprising an array of two or more rows andtwo or more columns of samples; and

an intra-image predictor configured to derive predicted samples of thecurrent region with respect to one or more of a group of referencesamples of the same image in dependence upon a prediction direction,defined by the selected prediction operation, between a current sampleto be predicted and a reference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

27. Apparatus according to clause 26, in which the intra-image predictoris configured to derive the predicted samples by interpolating amongstone or more sets of reference samples.

28. Apparatus according to clause 26 or clause 27, in which theintra-image predictor is configured to interpolate amongst two or moresets of reference samples to derive a respective intermediate samplevalue from each set of reference samples, and to combine theintermediate sample values to derive the predicted sample.29. Apparatus according to clause 28, in which each set of referencesamples comprises samples from a respective one or the two or moreparallel arrays of reference samples.30. Apparatus according to clause 29, in which each set of referencesamples comprises a set, in the respective array of reference samples,pointed to by the prediction direction.31. Apparatus according to clause 28, in which the intra-image predictoris configured to combine the intermediate sample values according to aweighted combination, in which a weighting applied to an intermediatesample value decreases with increasing separation of the set ofreference samples, from which that intermediate sample value asgenerated, from the current region.32. Apparatus according to any one of clauses 26 to 31, in which theintra-image predictor is configured to combine two or more sets ofreference samples to derive intermediate reference sample values, and toderive the predicted sample from the intermediate reference samplevalues.33. Apparatus according to clause 32, in which the intra-image predictoris configured to derive the predicted samples by interpolating amongstthe intermediate reference samples.34. Apparatus according to clause 32, in which each set of referencesamples comprises samples from a respective one or the two or moreparallel arrays of reference samples.35. Apparatus according to clause 34, in which each set of referencesamples comprises a set, in the respective array of reference samples,pointed to by the prediction direction.36. Apparatus according to clause 32, in which the intra-image predictoris configured to combine the reference sample values according to aweighted combination, in which a weighting applied to a reference samplevalue decreases with increasing separation of the set of referencesamples containing that reference sample value, from the current region.37. Apparatus according to clause 32, in which the intra-image predictoris configured to combine the two or more parallel linear arrays ofreference samples to form a linear array of reference samples.38. Apparatus according to any one of clauses 26 to 37, in which theselector is configured to select amongst two or more groups of referencesamples, each group comprising a respective different number of parallelarrays of reference samples.39. Apparatus according to any one of clauses 26 to 38, in which thecontroller is configured to detect encoded data identifying theprediction operation selected for each region of the image.40. Apparatus according to any one of clauses 26 to 39, in which:

the linear arrays of reference samples comprise one or more rows and oneor more columns of reference samples disposed with respect to thecurrent region; and

for a non-square current region of n rows and m columns of samples,where n does not equal m, for at least some of the candidate predictionoperations, the group of reference samples comprises differentrespective numbers of rows and columns of reference samples.

41. Apparatus according to clause 40, in which:

when n is less than a threshold value but m is at least the thresholdvalue, the set of candidate prediction operations comprises predictionoperations in which no more than a first upper limit of rows ofreference samples are used; and

when m is less than a threshold value but n is at least the thresholdvalue, the set of candidate prediction operations comprises predictionoperations in which no more than a second upper limit of columns ofreference samples are used.

42. Apparatus according to clause 40, in which:

when n is less than m, the set of candidate prediction operationscomprises prediction operations in which the number of columns ofreference samples is greater than the number of rows of referencesamples; and

when m is less than n, the set of candidate prediction operationscomprises prediction operations in which the number of rows of referencesamples is greater than the number of columns of reference samples.

43. Apparatus according to clause 40, in which:

when n is less than m, the set of candidate prediction operationscomprises prediction operations in which the number of rows of referencesamples is one and the number of columns of reference samples is greaterthan the number of rows of reference samples.

44. Apparatus according to clause 42, in which the first upper limit isequal to the second upper limit.

45. Apparatus according to clause 44, in which the first upper limit andthe second upper limit are equal to one.

46. Apparatus according to clause 41, in which the threshold value iseight samples.

47. Video storage, capture, transmission or reception apparatuscomprising apparatus according to any one of clauses 26 to 46.

48. An image encoding method comprising:

selecting, from a set of candidate prediction operations each definingat least a prediction direction, a prediction operation for predictionof samples of a current region of a current image, the current regioncomprising an array of two or more rows and two or more columns ofsamples; and

deriving intra-image predicted samples of the current region withrespect to one or more of a group of reference samples of the same imagein dependence upon a prediction direction, defined by the selectedprediction operation, between a current sample to be predicted and areference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

49. Computer software which, when executed by a computer, causes thecomputer to carry out a method according to clause 48.

50. A machine-readable non-transitory storage medium which storessoftware according to clause 49.

51. A data signal comprising coded data generated according to themethod of clause 48.

52. An image decoding method comprising:

selecting, from a set of candidate prediction operations each definingat least a prediction direction, a prediction operation for predictionof samples of a current region of a current image, the current regioncomprising an array of two or more rows and two or more columns ofsamples; and

deriving intra-image predicted samples of the current region withrespect to one or more of a group of reference samples of the same imagein dependence upon a prediction direction, defined by the selectedprediction operation, between a current sample to be predicted and areference position amongst the reference samples;

in which, for at least some of the candidate prediction operations, thegroup of reference samples comprises two or more parallel linear arraysof reference samples disposed at different respective separations fromthe current region.

53. Computer software which, when executed by a computer, causes thecomputer to carry out a method according to clause 52.

54. A machine-readable non-transitory storage medium which storessoftware according to clause 53.

The invention claimed is:
 1. An image encoding apparatus, comprising:processing circuitry configured to: select, from a set of candidateprediction operations each defining at least a candidate predictiondirection, a selected prediction operation for prediction of samples ofa current region of a current image, the current region comprising anarray of two or more rows and two or more columns of samples; derivepredicted samples of the current region according to reference samplesfrom a group of reference samples of the same image in dependence upon ashape of the current region and a selected prediction direction, definedby the selected prediction operation, between a current sample to bepredicted and a reference position amongst the reference samples, for atleast one of the candidate prediction operations in a case that thecurrent region is square, the group of reference samples comprising twoor more rows of reference samples disposed above the current region andtwo or more columns of reference samples disposed to a left of thecurrent region, and for at least another one of the candidate predictionoperations in a case that the current region is non-square, the group ofreference samples comprising either only one row of reference samplesdisposed above and adjacent to the current region or only one column ofreference samples disposed to the left of and adjacent to the currentregion; and encode the current region according to the predicted samplesof the current region.
 2. A video storage, capture, transmission orreception apparatus comprising apparatus according to claim
 1. 3. Theimage encoding apparatus as claimed in claim 1, wherein, for a secondother one of the candidate prediction operations in the case that thecurrent region is non-square, the group of reference samples comprisingone of (i) only one row of reference samples disposed above the currentregion and two or more columns of reference samples disposed to the leftof the current region and at different respective separations from thecurrent region, and (ii) only one column of the group of referencesamples disposed to the left of the current region and two or more rowsof reference samples disposed above the current region and at differentrespective separations from the current region.
 4. The image encodingapparatus as claimed in claim 1, wherein the two or more rows ofreference samples are adjacent to the current region and adjacent to oneanother, or the two or more columns of reference samples are adjacent tothe current region and adjacent to one another.
 5. The image encodingapparatus as claimed in claim 1, wherein at least two of the two or morerows of reference samples are not adjacent to each other, or at leasttwo of the two or more columns of reference samples are not adjacent toeach other.
 6. The image encoding apparatus as claimed in claim 1,wherein, far all of the candidate prediction operations in the case thatthe current region is square, the group of reference samples comprisesrespective sets of two or more rows or columns of reference samplesdisposed at different respective separations from the current region. 7.The image encoding apparatus as claimed in claim wherein, in the casethat the current region is non-square, the current region is defined bysplitting a codling unit.
 8. An image decoding apparatus, comprising:processing circuitry configured to: select, from a set of candidateprediction operations each defining at least a candidate predictiondirection, a selected prediction operation for prediction of samples ofa current region of a current image, the current region comprising anarray of two or more rows and two or more columns of samples; derivepredicted samples of the current region according to reference samplesfrom a group of reference samples of the same image in dependence upon ashape of the current region and a selected prediction direction, definedby the selected prediction operation, between a current sample to bepredicted and a reference position amongst the reference samples, for atleast one of the candidate prediction operations in a case that thecurrent region is square, the group of reference samples comprising twoor more rows of reference samples disposed above the current region andtwo or more columns of reference samples disposed to a left of thecurrent region, and for at least another one of the candidate predictionoperations in a case that the current region is non-square, group ofreference samples comprising either only one row of reference samplesdisposed above and adjacent to the current region or only one column ofreference samples disposed to the left of and adjacent to the currentregion; and decode the current region according to the predicted samplesof the current region.
 9. A video storage, capture, transmission orreception apparatus comprising apparatus according to claim
 8. 10. Theimage decoding apparatus as claimed in claim 8, wherein for all of thecandidate prediction operations in the case that the current region issquare, the group of reference samples comprises respective sets of twoor more rows or columns of reference samples disposed at differentrespective separations from the current region.
 11. The image decodingapparatus as claimed in claim 8, wherein at least two of the two or morerows of reference samples are not adjacent to each other, or at leasttwo of the two or more columns of reference samples are not adjacent toeach other.
 12. The image decoding apparatus as claimed in claim 8,wherein for all of the candidate prediction operations in the case thatthe current region is square, the group of reference samples comprisesrespective sets of two or more rows or columns of reference samplesdisposed at different respective separations from the current region.13. The image decoding apparatus as claimed in claim 8, wherein, in thecase that the current region is non-square, the current region isdefined by splitting a coding unit.
 14. An image encoding method,comprising: selecting, from a set of candidate prediction operationseach defining at least a candidate prediction direction, a selectedprediction operation for prediction of samples of a current region of acurrent image, the current region comprising an array of two or morerows and two or more columns of samples; deriving predicted samples ofthe current region according to reference samples from a group ofreference samples of the same image in dependence upon a shape of thecurrent region and a selected prediction direction, defined by theselected prediction operation, between a current sample to be predictedand a reference position amongst the reference samples, for at least oneof the candidate prediction operations in a case that the current regionis square, the group of reference samples comprising two or more rows ofreference samples disposed above the current region and two or morecolumns of reference samples disposed to a left of the current region,and for at least another one of the candidate prediction operations in acase that the current region is non-square, the group of referencesamples comprising either only one row of reference samples disposedabove and adjacent to the current region or only one column of referencesamples disposed to the left of and adjacent to the current region; andencoding the current region according to the predicted samples of thecurrent region.
 15. A non-transitory computer readable medium includingcomputer program instructions, which when executed by a computer, causethe computer to perform the method of claim
 14. 16. An image decodingmethod, comprising: selecting, from a set of candidate predictionoperations each defining at least a candidate prediction dire lion, aselected prediction operation for prediction of samples of a currentregion of a current image, the current region comprising an array of twoor more rows and two or more columns of samples; deriving predictedsamples of the current region according to reference samples from agroup of reference samples of the same image in dependence upon a shapeof the current region and a selected prediction direction, defined bythe selected prediction operation, between a current sample to bepredicted and a reference position amongst the reference samples, for atleast one of the candidate prediction operations in a case that thecurrent region is square, the group of reference samples comprising twoor more rows of reference samples disposed above the current region andtwo or more columns of reference samples disposed to a left of thecurrent region, and for at least another one of the candidate predictionoperations in a case that the current region is non-square, the group ofreference samples comprising either only one row of reference samplesdisposed above and adjacent to the current region or only one column ofreference samples disposed to the left of and adjacent to the currentregion; and decoding the current region according to the predictedsamples of the current region.
 17. A non-transitory computer readablemedium including computer program instructions, which when executed by acomputer, cause the computer to perform the method of claim 16.